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_ ^ Abstract 

We study the stability of receding horizon control for continuous-time non-linear stochastic differ- 
ential equations. We illustrate the results with a simulation example in which we employ receding 
horizon control to design an investment strategy to repay a debt. 



< 



Keywords: Receding horizon control, Stochastic differential equations, Stochastic optimal 
control, Hamilton-Jacobi-Bellman equations, Lyapunov functions, Ito's formula, optimal 
ON investment. 



1. Introduction 



u 
o 

In Receding Horizon Control (RHC), the control action, at each time t in [0, oo), is derived 
from the solution of an optimal control problem defined over a finite future horizon [t, t + T]. The 
RHC strategy establishes a feedback law which, under certain conditions, can ensure asymptotic 

'— 1 stability of the controlled system. This control strategy has been successfully developed over the 
last twenty years for systems described by deterministic equations. In this context RHC is also well 
J> known as Model Predictive Control (MPC) and has proven to be very successful in dealing with 

non-linear and constrained systems, see e.g. PQEIE]. The extension of RHC from deterministic to 
stochastic systems is the objective of current research. RHC schemes for the control of discrete-time 

CO stochastic systems have been proposed recently in [U [51 El E] . 

In this note, we discuss RHC for systems described by continuous-time non-linear stochastic 
differential equations (SDEs). To the extent of our knowledge, the RHC strategy has not yet been 
considered in this context. In order to study the stability of RHC for continuous-time SDEs, we 
formulate conditions under which the value function of the associated finite-time optimal control 
problem can be used as Lyapunov function for the RHC scheme. This is a well established approach 
for studying the stability of RHC schemes, which here is extended using Lyapunov criteria for 
stochastic dynamical systems [8]. We illustrate this contribution with a simple example of an 
optimal investment problem. Optimal investments problems are well suited to be tackled by 
stochastic control methods, see e.g. (9JHU]. In our example, we design an investment strategy 
to repay a debt. Having negative wealth due to an initial debt, the investor has the option to 
increase his/her current debt in order to buy a risky asset. The asymptotic stability of the adopted 
RHC scheme guarantees that the wealth of the investor tends to zero, so that the initial debt is 
eventually repaid. 
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2. Problem statement 



Let (Cl, T , P) be a complete probability space equipped with the natural filtration (Ftjt^Q 
generated by a standard Wiener process W : [0, oo) x $7 — > M. d on it. We consider a controlled 
time- homogeneous SDE for a process X : [0, oo) x Cl — > W 1 , 



dX, 
X, 



0,xo,u 
t 

0,xo,u 




b(X°' X0 ' u , u t )dt + a(X° t ^ u , u t )dW t , 



(1) 



xq, 

where x e R n ; b : W 1 x R m ->■ M n and a : W 1 x R m -> M rixd are measurable functions and satisfy 
|6(x,u)| + \a(x,u)\ < C(l + V(a;,«) £l"x J7, (linear growth), 

and 

|6(x,u) - b(y,u)\ + |<r(z,tt) - <r(y,ti)| < C\x - y\,V(x,y,u) G 1" x 1" x [/, (Lipschitz), 
for some constant C; and ur.\ is an admissible control process 



u (0 G 



:= |u : [0, oo) x — >■ £7 : progressively measurable and E ^ \ut(oj)\ 2 dt < ool 



with the set £/ C M m compact. Here the superscripts of X°' x °' u mean that the initial value of the 
process at time is xq and the involved control process is ur.y In this paper we are concerned 
with the conditions under which there exists a control process that drives the stochastic system 
X to the origin G M n and guarantees asymptotic stability of the controlled process. Here, the 
following definition of stability is adopted [5]: 



Definition 2.1. Given a stochastic continuous-time process X : M + x Cl 
[0,oo), with X = x G M n 



where 



(SI) The origin is stable almost surely if and only if, for any p > 0, e > 0, there is a S(p,e) > 
such that, if |xq| ^ 5(p,e), 



sup ^ e 



(SI') An equivalent definition to (SI) is: Let /i(-) : M+ — > M+ be a scalar-valued, nondecreasing, 
and continuous function of \x\. Let h(0) = 0, h(r) > for r / 0. Then the origin is stable 
almost surely if and only if, for any p > 0, A > 0, there is a 6(p, A) > such that, for 
|x | < 6(p,X), 



sup h(\X t \) ^ A 



P- 



(S2) The origin is asymptotically stable almost surely if and only if it is stable a.s., and Xt — > 
a.s. for all xq in some neighborhood R of the origin. If R = W 1 then we add 'in the large'. 
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3. Main results 

Let T > 0. As a preliminary step we consider the SDE for X : [t, T] X Q 
the point x G W 1 at the time t G [0, T] 



starting from 



Let / : M n x M. m — > M + and g : M n — > M+ be measurable nonnegative functions. Now we consider 
the problem of minimizing the following cost functional, V(i,x) G [0, T] x M. n , 



J[t, x; T; ur.-\] := E 



J fiXt^^ds + giX^) 



over the set IA of admissible control processes. We define the value function as 

cT 



v(t, x;T) := inf J[t, x; T; u.] = inf E 



(3) 



(4) 



and denote u*(t, x;T), t ^ s ^ T, the optimal control process if it exists. In particular, when t = 
we denote V(x; T) := u(0, x; T). 

Standard stochastic optimal control theories (see, for instance, I12j ) about the controlled 
SDE ([!]) tell us that the Hamilton-Jacobi-Bellman (HJB) equation for the value function v(-, -;T) 
is, V(t,x) G [0,T] x R n , 



inf 

ueu 



- d t v(t,x;T) 

v(T,x;T) = g(x). 

Hereafter we use the notations 

/ JhL 



ti[aa*{x,u)D 2 v{t,x;T)] + {b(x,u),Dv(t,x;T)) + f(x,u) 



(5) 



d 2 V 



d t v := —, Dv 
at 



and D 2 v 



\ 



dv 
Ox,-, 



/ d 2 v 

' dx{ dxidx 2 



d 2 v d 2 v 
\ dx„dxi 8x n dx2 



d 2 v \ 
ldx n 



dx 



d 2 v , 

dxl J 



Suppose this HJB equation has a solution and that the infimum in the equation is attained by 
u(t,x;T) for every (t,x) G [0,T] x R n , i.e., 



-dtv(t, x; T) = -tr[<7cr*(x, u(t, x; T))D 2 v(t, x; T)] + (b(x, u(t, x; T)), Dv(t, x; T)) + f(x, u(t, x; T)), 



then we construct the optimal control process for the SDE @ as 

u*(t,x;T) := u(s, Xf'*'"; T), t^s^T. 
As a consequence, the value function turns out to be 



(6) 



v(t,x;T) = E 



T 



f(Xr> u , U * s )ds + g(X% 



t,x,u* • 



= E 


[I 







t,X,U\ 



f(Xl^,u(s,X^;T))ds + g(X^ 



Now, for all the states x G W 1 all the time, we apply the specifically designed feedback law 

u c (x;T) :=u(0,x;T) (7) 

to the stochastic system 0. In other words, for the state x^' x °' u , at any time t #s 0, we apply 
only the initial optimal control 

u%X°' X0 ' uC ;T) = u(0,X^ Xo ' uC ;T) = u* (0, X^ '^; T) 

to the system. In particular, for t = 0, u c (Xq ,x °' u ; T) = u c (xo; T) = tt(0, xo; T). We call u c (-;T) the 
continuous receding horizon control process with the receding horizon T for the controlled SDE ([!]) . 
When t = 0, using the HJB equation and ([7]), we obtain 

-dtv(t, x; T)\t=o - f(x, u c (x- T)) = ^tr[aa*(x, u c (x; T))D 2 V(x; T)} + (b(x, u c (x; T)), DV(x; T)). 
Let us denote 

<j>{x;T) := dtv(t,x;T)\ t=0 + f(x,u c (x;T)) (8) 
for all x G W 1 . Let us assume that 

V £ C 2,1 (IR ra x (Al) 

i.e. the set of functions ip : W n x M + — > M, (x, i) i— > ip(x, t) that are twice continuously differentiable 
in x and once continuously differentiable in t; and that 

0(x; T) > 0, Vi £ 1" \ {0}, and 0(0; T) = 0. (A2) 

In particular, note that (p(0; T) = can be achieved if we suppose that 

6(0, 0) = 0, and <r(0, 0) = 0; (A2.1) 

and that 

/(0, 0) = 0, f(x, u) > 0, Vx G M n \ {0}, Vu G C7, and 5(0) = 0, g(x) > 0, Vx G M n \ {0}. (A2.2) 



We now state one of our main results and then discuss Assumptions (Al A2) in more detail. 

Proposition 3.1 (Convergence). Suppose &(•,•) and <r(-,-) are of linear growth and Lipschitz. 
Suppose the HJB equation ^ has a unique classical solution. Under the assumptions (Al), and 
(A2), we have that almost all the trajectories of the stochastic system ([!]) driven by the continuous 
receding horizon control u c (-;T) defined in Q converge to the origin. 

Proof. Temporarily we denote Xt := X^' x °' u for simplicity. By Ito's formula and the Hamilton- 
Jacobi-Bellman equation we have 

dV(X t ;T) = (DV(X t ;T),dXt) + ^tr[aa*(Xt,u c (X t ;T))D 2 V(Xt;T)]dt 

= {DV(X t ;T), b(X t , u c (X t] T)))dt + {DV{X t - T),a(X t , u c (X t ;T))dW t ) 

+ ^tr[aa*(X t , u c (X t ; T))D 2 V(X t ; T)]dt 
= -<f>{X t - T)dt + (DV(X t ;T),a(X t ,u c (X t ; T))dW t ). 
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Therefore by the assumption (A2) we get that, for ^ s ^ t < oo, 

ft 



E[V{X t] T)\T s }-V(X s ;T) 



-E 



< 0. 



(9) 



In particular when s = 0, E [VpCtjT)] < V(X ;T) = V(x ;T) < oo. Hence, {V(X t ;T), T t }o^t<oc 
is a nonnegative supermartingale. Let us recall the supermartingale convergence theorem, see, for 
instance, [131 pag. 18]: Suppose {Mj, ^i}o<*<oo is a right-continuous, nonnegative supermartingale. 
Then M^u;) := lim^oo Mf(uj) exists for P-a.e. w £ fi, and {Mj, .Ft}o<t<oc is a supermartingale. 
Hence, there exists a random variable V^, integrable, such that V(Xf,T) — > Vx>, as.. Now, 
equation (|9|) implies particularly that 

/•oo 

(10) 



E [V^ - V(x ; T) = — E [cf>(X t ;T)] dt, 

Jo 

which in turn gives lim^oo E [cj)(Xt; T)] = 0. Therefore we obtain that 

lim 4>(Xt;T) = 0, a.s., 



t— >oo 



because 0^0. Denote the set $ := {x G M n : (p(x;T) = 0} and from the assumption (A2) we 
know that $ = {0} and it is closed. Thus by the continuity of <p we learn that almost all the 
trajectories of the process Xt converge to the origin in W 1 . □ 



In ( Al ), continuous twice differentiability of V is required for the applicability of Ito's formula. 
In order to illustrate Assumption (A2) note that, since ^ is time-homogeneous, we have 

^(t,x;T)| t= o = -d H v(0,x;H)\ H=T = -d H V(x; H)\ H=T . 

Hence, <f>{x\ T) can be equivalently expressed as: 

T) = -d H V(x; H)\ H=T + f(x, u c (x; T)). (11) 

Note that, here, dnV(x; H)\h=t is the rate at which the optimal cost increases with increments 
in the horizon T. Note that (f>(x; T) > is implied by 



d H V(x;H)\ H=T ^0. 



:i2) 



Here, we recover the condition of monotonic decrease of the value function with the length of the 
horizon, which is a well-studied condition for the stability of receding horizon control schemes in 
a deterministic setting, see e.g. [21 [THE]. Finally, note that Assumption (A2.1) requires that the 
state, at the origin, is not affected by the noise. This is an unavoidable assumption for obtaining 
asymptotic stability in the sense of Definition 2.1 If this assumption is not met, then one has to 
resort to other notions of stability such as, for example, mean square boundeness, see e.g. [H [71 [T6] , 
Under additional assumptions, asymptotic stability according to Definition 2.1 can be obtained. 
Let us assume in addition that for any e > 0, there exists a 5(e) > such that 



V{x;T)<e if \x\ ^ 6(e); 
and that there exists a continuous, nondecreasing function h : I 

h(0) = 0, h(r) > 0,Vr / 0, 

such that 

V(x;T) > h(\x\), Vx G R n . 



(A3) 



satisfying 



(A4) 
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Corollary 3.2 (Asymptotic stability). Suppose &(■, •) and <r(-, •) are of linear growth and Lip- 
schitz. Under the assumptions (Al), (A2), (A3), and ( |A4[ ), we have that the origin in M. n is 
asymptotically stable. 

Proof. Let us recall the supermartingale inequality, see, [131 P a g- 13]: Suppose {M t , Tt}o^t<oo is 
a supermartingale whose every path is right-continuous. Let A > and [a, b] be a subinterval of 
[0,oo). Then, we have 

E[M+] 



sup M t > A 

a<t<b 



A 



Hence, for any A > 0, we have 



sup V(X t ;T)>\ 

_0^t<oo 



E[V(x -T)} V(x ;T) 



A 



A 



Using assumption (A3), we obtain that for any A > 0, p > 0, there exists 5(p, A) > such that 

sup V(X t ;T)^X ^p if \x \ < S(p, A). 

0<t<oo 



Then, by assumption (A4), we immediately obtain 

£ I 



sup h(X t ) ^ A 

(Kt<oo 



sup V(X t ;T)>\ 

0sgi<oo 



□ 



which entails the asymptotic stability of the origin in M n . 

In the following section, we present a simple illustrative example in which Assumptions (Al A4) 
can be verified explicitly. 



4. Example: repayment of a debt 

In this example we consider a variant of the Merton's portfolio problem, see e.g. I12j . 
Suppose in a complete financial market there are only two assets, one asset being risk free such as, 
for instance, a bank deposit or a bond, and the other one being a risky asset such as, for instance, 
a stock. The assets obey the following price process, t ^ 0, 

dpi = rP?dt, (13) 
dP? = bP?dt + aP?dW t , 

where b > r > 0, and a ^ are given constants. Here r is the risk free interest rate for the bank 
deposit, b is the drift, or average, rate of the stock's return, and a is the volatility of the stock's 
return. Suppose an agent's total wealth at time t is Xt, comprising the risk free part n|P t and the 
risky part n^P t 2 , where n| and n 2 are the quantity of the assets, respectively. The control variable 
ut is the portion of the total wealth Xt invested by the agent on the risky asset at time t. In a 
continuous-time setting, it is assumed that the allocation of wealth takes place instantaneously. 
Hence, for t #J 0, we have: 

x t = njpi + nfif, 

H]Pl = (l-u t )X t , (14) 

nfp t 2 = u t x t . 
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We assume that the investment obeys the self financing condition. That is, starting from an 
initial wealth xq, the agent can only sell or buy these two assets but is not allowed to borrow 
money from outside or consume his or her wealth. Written down as a differential equation, this is 



P/dllJ + Pfdli 2 t = 0. Using Q and Q we obtain 

dX t = Tl\dP} +Il 2 t dP? + P}dYl\ + P?dYl 2 t 



= [r + (b - r)u t ]X t dt + au t X t dW t . 
Therefore, the wealth satisfies the stochastic differential equation 

dX o,x ,u = [ r+ ^_ r ) Ut i l x^ u dt + au t X^ X0 ' u dW t , (15) 

A — Xq, 

which is called the wealth process. 

In this example, we assume that the agent's initial wealth is negative and his or her aim is to 
repay any debt eventually. Hence, we have xq < and investigate the asymptotic stability of the 
origin 6 1. In this case, the wealth process will always be negative until the time it reaches zero. 
However, note that Xt < does not necessarily mean that the agent has only debt without any 



money to invest or stock to sell. If negative values of ut in (14) are allowed then the investor is 
able to increase his or her debt in order to buy stocks. Let us explain the financial meanings of all 
possible values of Ut when Xt < 0: 

• If ut = 0, then n^P^ = Xt and n 2 P 4 2 = 0. In this case, the investor just owns a debt with 
the bank equal to his or her negative wealth. 

• If u t < 0, then UjP^ < X t and n 2 P t 2 > 0. In this case, the investor is borrowing additional 
funds from the bank and is using it to buy stocks. The investor owns a debt with the bank 
equal to (1 — ut)Xt and a positive quantity of stocks whose value is UtXt- 

• If ut > 0, then n|P/ > Xt and n 2 P 2 < 0. In this case, the investor is borrowing stocks 
(short selling) and is using this additional wealth to reduce his or her debt with the bank. 
However, in general, this is not desirable because a debt in stocks is more risky than a debt 
with the bank. 

Note that if n^P/ < (i.e. when the investor has a debt with the bank) then r is the rate at which 
interests are payed to the the bank when owing debt. Here, in order to simplify the exposition of 
the problem, we assume that r is the same whether n^P/ < or n^P/ > 0. However, it will be 
shown that this issue is immaterial. In fact, the derived control process will be constantly negative 
until the wealth reaches zero. In turn, this means that n^P/ < and, therefore, r will not change 
meaning throughout. Finally, we consider the constraints ut G [c±, C2] with c\ < and C2 #s 0. The 
constraint c\ means that the investor is not allowed to increase the debt with the bank by more 
than (1 + |ci|) times his or her current (negative) wealth. Similarly, C2 is a constrain on borrowing 
stocks. If short selling is not allowed then c% = 0. 

4-1. Running cost 

We consider cost function ^ with 



f(x,u) := < ' ^ and g(x) := 0, Vcc, (16) 




with (3 > 2 being a given constant. In this case, the value function v(t,x;T) satisfies the HJB 
equation, (t,x) £ [0,T] x (— oo,0], 



- d t v(t,x;T) 

v(T,x;T) = 0. 



inf 

m£[ci,c 2 ] 



1 



aux) 2 D 2 v(t, x; T) + (r + (b — r)u)xDv(t, x; T) + (— x) 



117) 



If D 2 v(t, x;T) > then a necessary condition for u to be a minimiser is 



o- 2 MX^L>^(i, x; T) + (b - r)xDv(t, x; T) = 0, 



that is, 



u = — - 



(b-r)Dv(t,x;T) 
a 2 xD 2 v(t,x;T) 



In order to solve the HJB equation (17), we try to find a value function in the form 



v(t,x;T) 



(-x)Pw(t), x^O, 
0, x > 0, 



with w to be determined. For x ^ 0, by substituting (19) into (18), we obtain 

(6-r)(-/3)(-x) /3 - 1 b-r 



a 2 x{3{(3 - l)(-x)P- 



(18) 



(19) 



(20) 



Note that the so-obtained u has the same expression as in the classical Merton's problem (although 
here we assumed /3 > 2 instead of (3 < 1), see e.g. [HI pag. 160-161], or [12J P a g- 168-169]. Using 
(19) and (20), the HJB equation becomes 

x \- x f- 2 w {t) - (fir - ^J"^ ) (-xf^xwit) + {-xf 
/3(6-r) 2 



-{-xfw'{t) 



P(b-r) 2 
2(/3-lV 2 ' 



that is, 



w'(t) 

and we obtain that the solution is 
where 



2((3-l)a 2 



fir 



w(t) - 1 



W (t) = -(l-e^-' n 



P(b~r) 2 
V ■= ~ P r - 



■ 2((3-l)a 2 



Hence, the value function ( 19 ) is actually given by 



v(t,x;T) 



'(- x fl(l-e^- T )), x^0, 
0, x > 0. 



(21) 

(22) 
(23) 

(24) 



Here we assume r] ^ 0. Note that whether rj < or ij > it always holds that w(t) > 0. Thus 
D 2 v = p(/3 - \){-xf- 2 w{t) > as we expected. (However, in the next subsection, we will narrow 
the requirement to be r] > 0.) 



Eventually, we obtain that the corresponding receding horizon control process is 



u c (X t ;T) = u(0,X t ;T) 



The stochastic system (15) under the receding horizon control process (25) becomes 

(b-r) 2 



dX. 



X, 



t 

0,XQ,U 





(/3 - IW 



x 



0,xq,u 



dt 



(/3 - 1)<7 



X, 



t 



dW t , 



(25) 



(26) 



XQ. 



In this case the solution turns out to be a geometric Brownian motion that can be written down 
explicitly (see e.g. [13, pag. 349-50]) 



X 



0,xq,u 
t 



xq exp 



(2/3 - 



2(/3- 1)V 2 



(P-l)a 



W t 



(27) 



The asymptotic properties near the origin can be seen directly from the explicit solution (27). 
However, in the following subsection we will use Proposition 3.1 and Corollary |3.2| to asses the 
stability of the system. Note that the derived control policy ( 25 ) is in fact a constant process with 
negative value. This implies that the agent is always advised to borrow money from the bank and 
buy stocks with it. He or she will repay the debt in the end when his or her total wealth reaches 
zero by making profit from investment in stocks. 

4-2. Verification of assumptions 



Here we verify when Assumptions (Al A4) are met by the receding horizon control process (25). 
Note that, for v{0, x; T) given by Q we have V(x; T) = v(0, x; T) < oo for all (x, T) G R n x R + . 
In addition, since /3 > 2, we learn that 



D 2 V(x;T) 



' /3{p-l){-x)P- 
0, 



(l - e~ vT ) , x ^ 0, 
x > 0, 



is continuous. This, combined with the continuity of 

d T V(x;T) : 



(-x) /3 e" r ' T , x < 0, 
0, x > 0, 



ensures that V G C ' (. 



2,1 /inn 



-,K); thus Assumption (Al) is satisfied. For 



(x) =d t v(t,x;T)\ t=0 + f(x,u c (x;T)) 



\-xf{l-e 
0, 



x>0. 



to be positive when x < we require rj > 0. Hence Assumption (A2) is satisfied when rj > 0. In 
light of the continuity of V we know that for any e > there is 5 > such that V(x; T) < e for 
— 5 < x < 0; thus Assumption (A3) is satisfied. For x\ < xi < we have V{x\;T) > V{x2\T) > 
thus Assumption (A4) is satisfied if we choose h to be V(-;T) itself. 
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In conclusion, for given r, b and a, we obtain that Assumptions ( A1|A4 ) are satisfied for all 
choices of ft such that: 

ll(6-r) 2 



2 < ft < 1 + 



(28) 



2r a 2 ' 

where the inequality on the right-hand side corresponds to the condition rj > 0. Thus, a necessary 
condition to have a stabilizing control process is that the right-hand side of the above inequality is 
greater that the left-hand side; that is (b — r) 2 > 2ra 2 . Finally, it is easy to se that the constraint 
u £ [ci,C2] can be met provided that it is possible to choose 



ft > 1 + 



(b-r) 

\Cl\(T 



(29) 



which, taking into account (28), can be done if |ci| > 2r/(b — r). 

4-2.1. Terminal cost 

It is also possible to consider a cost function in the form 



f(x,u) := 0, Vx, and g(x) :- 



0, x > 0, 



(30) 



The derived controlled process turns out to be the same as in (25). However, in this case, the value 
function is given by 



v(t,x;T) 



\_ x f e v(t-T)^ x ^ 0i 
0, x > 0. 



(31) 



where rj is given again by (23). Through similar steps, one eventually obtains the same stability 
conditions of the previous case. 



4-3. Numerical illustration 

We present a simulation example where: r = 0.03, b = 0.1, a = 0.15, and xq = —100. 
Here we illustrate the behaviour of the wealth process under the receding horizon control process 
(25) for three different choices of ft in cost function (16): ft = 2.1, ft = 4.5 and ft = 8. The corre- 
sponding Monte Carlo simulations are displayed in Figures [T]{3] respectively. Note that, according 
to (28), for the given values of r, b and a, we have that the wealth process is asymptotically stable 
for ft S (2, 4.6). By inspecting the figures, it can be seen that for ft = 2.1 the wealth process is 
clearly asymptotically stable but there is a significant risk of a large initial undershoot. For ft = 4.5 
the process converges much slower but the risk of a initial undershoot is reduced. For ft = 7.8 the 
wealth process is not asymptotically stable (note the different time scale in the figure). 
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Figure 1: Wealth process for /3 = 2.1 (100 simulations) 




-250 - 

-300 - i 

-350 - 

_400 1 i I i i i i I i i i 

10 20 30 40 50 60 70 80 90 100 

Time 

Figure 2: Wealth process for /3 = 4.5 (100 simulations) 
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Figure 3: Wealth process for ft — 7.8 (100 simulations) 



5. Conclusions 



In this note, we have discussed the RHC strategy for systems described by continuous-time 
SDEs. We have obtained conditions on the associated finite-horizon optimal control problem 
which guarantee the asymptotic stability of the RHC law. We have shown that these conditions 
recall their deterministic counterpart. We have illustrated the results with a simple example in 
which the RHC law can be obtained explicitly. In current work, we are addressing the problem of 
implementation in more realistic applications. For this purpose, it will be necessary to formulate 
conditions on the control problem ^ which can guarantee that Assumptions (Al A4) hold true 



and which can be imposed or verified easily. For problems where the dimension of the state space 
is not prohibitive, it is possible to solve the associated finite-horizon optimal control problem with 
numerical methods |17t [TH| . The other possibility is is to approach the finite-horizon problem ^ by 
direct on-line optimization. In this case, one considers a parameterized class of feedback policies 
and iteratively optimizes the parameter of the feedback policy, at regular time intervals, condi- 
tioned on the value of the current state. In this context simulation-based optimization methods 
have shown to be promising tools, see e.g. (T9l \20 [ [2"T]. 
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